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I.  SUMMARY 

1.  Purpose 

This  experiment  deals  with  the  effect  of  Increasing  the 
work  load  on  the  output  of  a  single  operator  and  PPI 
console.  '.Vbrk  load  was  increased  by  increasing  the  number 
of  targets  on  the  scope  face.  The  information  obtained  in 
thie  study  is  a  ne^^sary  preliminary  to  the  evaluation 
of  the  Cadillac  III  JIC  system.  A  rationale  for  the 
evaluation  of  thi3  system  is  presented. 

2.  Data 


The  data  for  this  report  were  obtained  under  laboratory 
conditions  which  simulated  the  Cadillac  III  CIC  system 
in  a  P0-1VV  airplane.  The  targets  were  presented  by  means 
of  an  electronic  target  generator.  The  data  were  gathered 
from  photographs  of  the  scope  picture,  recordings  of  the 
operators*  verbal  reports,  and  direct  observation  of 
plotting  behavior.  The  four  experimental  subjects  were 
naval  officers  with  CIC  experience.  Each  subject  was 
tested  four  times  on  four  equivalent  problems  under  con¬ 
trolled  conditions, 

3.  Conclusions 


a.  It  was  found  that  detection  time  for  a  new  target  was 
a  positive  exponential  function  of  the  number  of  tar¬ 
gets  on  the  scope  face.  With  no  targets  on  the  scooe, 
the  detection  time  for  a  new  target  was  found  to  be, 
on  the  average,  0.10  minutes,  with  14  targets,  1.35 
minutes. 

b.  The  chief  findings  concerning  errors  of  estimation 
are  given  in  the  table  on  the  following  page. 

Accuracy  of  direct  estimation  of  target  characteris¬ 
tics  is  given  in  terms  of  relative  and  absolute  errors. 
Errors  considered  with  respect  to  both  magnitude  and 
direction  are  called  relative  errors.  Errors  con¬ 
sidered  solely  with  respect  to  magnitude  are  called 
absolute  errors.  The  mean  relative  error  indicates 
constant  errors  of  estimation.  The  standard  devia¬ 
tion  (3.D.)  of  the  relative  errors  indicates  the 
consistency  of  the  estimates.  The  median  absolute 
error  indicates  the  accuracy  of  the  estimates  in 
terms  of  the  absolute  error. 
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n  A\naTnmfmTA  v 


Target 

Characteristic 

Relative 

Error 

Absolute 

Error 

Bearing 

Mean 

0.53 

Median* 

1.76 

(degrees) 

s.:>. 

2.58 

Q3-Q1  ** 

2.16 

Range 

Mean 

1.85 

Median 

2.40 

(miles) 

S.D. 

2.17 

0-3“  ^1 

2.10 

Course 

Mean 

1.13 

Median 

5.83 

(degrees) 

S.D. 

11.95 

^3-^1 

14.06 

Speed 

Mean 

130 

Range 

0-350 

(knots) 

*The  nedian  error  represents  the  mid-point  above  and 
below  which  50  percent  of  the  errors  were  observed. 

'♦Qg-qi,  the  interquartile  range,  represents  the  range 
of  the  middle  50  percent  of  the  observed  errors. 


c.  When  the  number  of  reports  made  within  a  minute 
interval  was  considered  as  a  function  of  the  run¬ 
ning  time  of  the  experiment,  a  oyolioal  relationship 
appeared;  i.e.,  about  10  to  12  reports  were  made 
per  minute.  It  was  also  found  that  the  time  between 
successive  reports  on  a  given  target  increases  with 
the  number  of  targets  to  be  reported. 

d.  The  number  of  plots  made  per  unit  time  remained  con¬ 
stant  under  varying  loads.  The  average  number  of 
plots  was  found  to  be  5.26  per  minute. 

l 


-2- 


t 


II.  SYSTEMS  EVALUATION 

In  the  area  of  human  engineering  the  concept  of  systems  re¬ 
search  is  relatively  new.  Definite  principles  of  systems  research 
are  still  being  evolved  in  the  daily  work  of  this  and  other  lab¬ 
oratories.  In  the  sense  considered  here,  a  "System"  is  thought 
of  as  a  functional  organization  of  men  and  machines  for  the 
accomplishment  of  some  task.  In  more  specific  terms  the  systems 
to  be  studied  here  constitute  a  functional  organization  of  the 
CIC  personnel  and  equipment  to  effect  the  various  stated  missions 
of  an  airborne  CIC. 

1.  The  Functional  Elements  of  a  Cl;  System 

Before  proceeding  to  the  discussion  of  how  the  CIC  sys¬ 
tem  may  be  evaluated,  the  functional  aspects  of  a  CIC  system 
should  be  considered.  These  elements  may  be  considered  as: 

Input  -  CIC  System  -  Output. 

The  input  into  the  system  inoludes  the  following:  (1) 
the  intelligence  briefing  and  the  operational  orders  given  to  the 
CIC,  (2)  information  obtained  through  radio  communication  from 
command,  combat  air  patrols  and  liaison,  and  (3)  the  radar  in¬ 
formation  obtained  from  the  scope  picture. 

The  output  of  the  CIC  system  includes  the  radio  and 
video  reports  to  command,  to  combat  air  patrols,  to  surface  units 
and  other  liaison  links.  These  reports,  which  have  been  digested 
and  integrated  by  the  CIC  officer  from  a  tactioal  point  of  view, 
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oontain  certain  information  as  to  the  number,  position,  speed  and 
identity  of  the  targets. 

The  CIC  system  consist?  basically  of  the  CIC  personnol 
with  their  respective  skills  and  capacities,  and  certain  special¬ 
ized  CIC  electronic  gear.  The  functions  of  the  individuals  with¬ 
in  the  system,  when  they  are  finally  evolved  and  defined,  will 
constitute  the  operational  procedure  and  doctrine  of  the  CIC. 

The  complex  system  may  be  conveived  as  comprised  of  a  number  of 
simple  sub-systems.  A  sub-system  is  a  man-machine  or  a  man-man 
link.  When  all  th6  possible  linkages  in  the  CIC  are  considered 
operating  as  a  single  unit,  the  complexity  of  the  CIC  system  con 
be  appreciated.  Before  the  total  system  can  be  understood,  the 
capacities  and  potentialities  of  these  sub-systems  must  be  known. 
In  this  study  a  particular  nan-machine  link  is  considered. 

The  total  system  may  be  capable  of  a  number  of  functions. 
These  functions  are  the  CIC  missions.  The  CIC  system  has  been 
envisaged  as  capable  of  the  following  missions:  airborne  early 
warning,  direct  air  interception,  direct  anti-submarine  warfare, 
friendly  submarine  coordination,  direct  missile  interception, 
strike  direction,  radar  counter  measures,  PO  link,  communication 
relay,  convoy  control  and  coordination,  air-sea  rescue  coordina¬ 
tion,  weather  reconnaissance,  and  possibly  others.  For  the  pur¬ 
poses  of  our  evaluation,  we  will  Unit  our  research  to  the  otudy 
of  A3W  and  air  strike  intercept,  which  have  been  conoid -’rsi,  '"o 
date,  as  the  primary  CIC  functions. 
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2.  A  Method  of  Evaluation 


A  method  of  evaluating  a  CIC  system  may  be  now  considered. 
The  evaluation  of  different  CIC  systems  may  be  formulated  in 
terms  of  a  comparison  of  the  effectiveness  of  alternate  functional 
CIC  systems  in  handling  given  problems  according  to  certain  estab¬ 
lished  criteria  of  success.  This  means  that  the  CIC  system  will 
be  presented  with  a  series  of  standardized  problems  and  the  re¬ 
sponses,  or* output,  will  be  observed.  This  output  may  be  evalu¬ 
ated  according  to  certain  criteria.  The  criteria  to  be  used  in 
our  studies  were  established  after  much  discussion  with  tactically 
experienced  consultants;  they  are  to  be  time,  accuracy,  amount 
and  quality.  It  can  readily  be  appreciated  that  the  system  which 
can  turn  out  the  largest  number  of  reports,  with  the  greatest  ac¬ 
curacy,  in  the  shortest  time,  and  of  a  superior  quality  for  tac¬ 
tical  purposes,  may  be  considered  to  be  the  best  system.  The  com¬ 
plex  of  man-machine  links  and  operating  procedures  which  consti¬ 
tutes  the  CIC  system  may  be  changed  in  various  ways.  Evaluating 
a  CIC  system,  then  means  the  testing  of  several  alternate  CIC 
systems  against  a  series  of  standardized  problems  to  compare  the 
output  from  these  alternate  systems  against  objective  criteria. 

This  orientation  implies  three  main  areas  of  research. 

The  first  involves  the  development  of  standardized  tactical  problems 
by  means  of  which  the  alternative  CIC  systems  may  be  tested.  The 
second  involves  the  development  of  alternative  linkages  and  operat¬ 
ing  procedures  for  the  P0-1V/  CIC  system.  The  third  phase 
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Involves  comparing  the  performance  outputs  of  the  alternative 
systems  according  to  specific  criteria.  It  Is  with  the  problems 
which  arise  in  the  first  area  that  this  report  is  concerned. 

3.  The  Need  for  Basic  Data 

This  study  deals  with  the  first  area  of  research  des¬ 
cribed,  namely,  problem  difficulty.  Problem  difficulty  Involves, 
in  the  main,  a  statement  concerning  the  total  load  which  le  placed 
on  a  given  system.  Target  load  may  be  broadly  defined  In  terms 
of  two  sets  of  variables:  (1)  space  density,  and  (2)  time  density. 
Space  density  refers  to  the  distribution  of  the  targets  over  the 
scope  face.  Time  density  involves  the  total  number  of  occurrences 
per  unit  time,  including:  (a)  the  new  targets  which  come  on  the 
scope,  (b)  the  fading  and  reappearance  of  old  targets,  (c)  target 
course  changes,  and  (d)  the  appearance  or  non-appearance  of  IFF 
signals.  It  Is  apparent  that  the  work  load  on  the  operator  In¬ 
creases  with  the  number  of  targets  which  must  De  considered  and 
with  the  numoer  of  events  that  occur  per  unit  time. 

This  experiment  deals  essentially  with  the  first  vari¬ 
able,  space  density,  which  Is  considered  here  in  terms  of  the  num¬ 
ber  of  targets  on  the  scope.  In  studying  this  parameter,  It  is 
necessary  to  work,  at  first,  with  a  single  sub-system.  This 
means  studying  the  capacities  and  potentialities  of  an  individual 
air  control  officer  and  his  associated  equipment.  The  researches 
are  expected  to  yield:  (1)  the  fundamental  data  on  sub-system 
capabilities,  and  (2)  the  necessary  data  for  obtaining  some 
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concept  of  the  difficulty  of  a  problem  in  terms  of  space  densi¬ 
ties. 

The  above  theoretical  foi-mula tion  furnishes  a  frame 
work  about  which  a  series  of  experimental  researches  are  being 
planned.  The  results  of  the  first  of  these  researches  are  pre¬ 
sented  in  the  remainder  of  the  report.  It  is  expected  that  the 
data  derived  from  this  and  subsequent  researches  will  furnish 
the  basic  information  necessary  for  a  more  complete  system 
evaluation. 


III.  THE  EXPERIMENTAL  STUDY 

The  theoretical  discussions  concerning  the  over-all  systems 
study  indicated  the  need  for  certain  kinds  of  information.  The 
initial  objective  is  to  obtain  data  concerning  the  capabilities 
of  the  various  sub -systems,  and  second,  the  factors  which  effect 
problem  difficulty  and  equivalence.  Throughout  this  report  the 
approach  to  the  limit  of  some  capability  of  a  sub-system  is  ?e- 
ferred  to  as  "saturation."  Four  factors  are  studied  with  re¬ 
spect  to  saturation  limits.  The  experiment  was  designed  to  ob¬ 
tain  fundamental  data  concerning  the  relationship  which  obtains 
between  space  density  and  (1)  the  detection  time  of  new  targets, 
(2)  the  number  of  reports  of  target  information  which  could  be 
made  in  a  given  unit  of  time,  (3)  the  accuracy  of  direct  unaided 
visual  estimation  of  range,  bearing,  course,  and  speed  of  targets, 
and  (4)  the  frequency  with  v/hich  the  position  of  any  target  is 
marked.  The  measure  of  space  density  used  in  this  study  is  the 
number  of  targets  already  on  the  scope  face  when  the  new  target 
appears . 

1 .  Description  of  Apparatus 

The  experiment  was  conducted  in  a  mock  up  of  the 
Cadillac  III  airplane.  This  full  scale  mock  up  simulates  the 
part  of  the  hull  of  a  P0-1W  airplane*  into  which  the  essential 
CIC  electronic  equipment  of  Cadillac  III  has  been  incorporated. 

The  experiment  was  conducted  on  the  IP-48  unit  of  the  APA/56 

^Manufacturer * s  Brief  Model  Specifications  for  the  Lockheed 
Constellation  Lfodel  749  CIC  Airplane;  Lockheed  Aircraft  Coro., 
Report  No.  6569,  June  11,  1948  (Confidential). 
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remote  radar  indicator.* 

The  IP-48  is  a  12"  PPI  repeater  radar  scope  on  which  a 
number  of  different  presentations  may  be  obtained.  For  the  pur¬ 
poses  of  this  experiment,  the  controls  were  so  set  that  the  sweep 
length  represented  a  200  mile  range,  and  the  scope  presented  a 
polar  grid  with  10  mile  range  rings  and  30°  angle  markers.  Video 
gain  was  adjusted  by  the  subjects  to  their  personal  preference. 

The  hull  was  darkened  to  simulate  actual  operating  conditions. 

% 

2.  Target  Simulation 

In  order  to  study  the  Cadillac  III  system  in  the  labora¬ 
tory,  it  was  necessary  to  develop  a  method  of  electronically  simu¬ 
lating  the  radar  information  which  is  ordinarily  fed  into  the  radar 
indicator.  The  Aircraft  Target  Generator  15-Ali-l**  was  developed 
for  this  purpose. 

Several  target  generators  were  used  to  provide  the  tar¬ 
gets  used  in  our  study.  They  enabled  the  setting  in  of  targets 
at  some  initial  position;  and  they  also  allowed  these  targets  to 
assume  a  pre-set  course  and  speed. 

A  script  was  prepared  beforehand  to  facilitate  the  run¬ 
ning  of  each  problem.  The  various  manipulations  necessary  to  simu¬ 
late  the  problems  were  listed  in  a  tine  sequence. 

♦Handbook  of  Maintenance  Instructions  for  Indicator  Assembly 
All/ APA-56  (XN-1);  Bureau  of  Aeronautics,  August  1,  1949,  (Confi- 
(fential ) . 

♦♦Target  Simulator  for  Cadillac  Evaluator:  Device  15-AM-l ;  O.N.R., 
Special  Devices  Center,  Report  No.  42  (N8-2),  November  14,  1949, 
(Confidential ) . 
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Ab  much  Information  as  poesible  was  pre-set  Into  the  simulator. 
After  the  Btart  of  each  problem,  It  was  necessary  only  to  make 
those  adjustments  which  were  Indicated  by  the  script. 

3.  Instructions  to  Subjects 

The  subjects  were  told  that  targets  would  appear  at 
random  positions  and  times  on  the  scope  face  and  follow  randomly 
selected  courses.  The  subjects  were  Instructed  to  report  Into  a 
voice  recorder  the  appearance  of  each  new  target  as  soon  as  It  was 
detected.  This  detection  report  was  to  be  given  priority  above 
any  other.  The  range,  bearing,  course  and  speed  of  the  target 
were  to  be  reported  as  frequently  as  possible.  All  Judgments 
with  respect  to  range,  bearing,  course  and  speed  were  to  be  made 
by  direct  visual  estimation,  utilising  only  the  10  mile  range 
rings  and  30°  angle  marks.  The  frequency  of  plotting  was  to  be 
maintained  at  the  subject's  own  option  or  convenience. 

4.  Description  of  Experimental  Problems 

In  the  construction  of  this  experimental  problem,  con¬ 
sideration  was  first  given  to  the  practical  tactical  experience 
of  airborne  CIC  personnel.  On  the  basis  of  thl6  Information,  It 
was  decided  that  a  thirty  minute  problem  Involving  fifteen  tar¬ 
gets  might  be  expected  to  tax  the  capabilities  of  the  scope  op¬ 
erators  very  heavily.  The  targets  were  introduced  at  Irregular 
intervale  of  30  seconds  to  3  minutes.  The  first  five  targets  were 
Introduced  within  the  first  two  minutes  in  order  to  build  up  the 
load  quickly.  The  initial  positions  of  the  targets  were  randomly 
chosen  points  on  the  scope  face.  This  does  not  simulate  usual 


tactical  problems,  but  it  has  the  advantage  of  permitting  broader 
generalization  from  our  conclusions.  For  the  sane  reason,  the 
courses  and  aDoeds  of  the  various  problems  were  chosen  at  random. 
Speeds  varied  between  100  and  600  knot 3.  Course  varied  through 
360° »  being  necessarily  limited,  however,  to  such  courses  which 
would  keep  the  target  on  the  scope  face.  The  target  tracks  in¬ 
volved  no  course  changes.  Some  idea  of  how  the  problem  appeared 
to  the  experimental  subject  may  be  gained  from  Fig.  1.  Table  1 
gives  the  initial  position,  bearing,  speed  and  course  of  each 
target. 


Table  1 

Target  Characteristics 


Target  No. 

Initial  Position 
(miles  from  center) 

Course 

(degrees) 

Speed 

(knots) 

Time  of 
Appearanoe 
(min.  :  sec.) 

1 

336  W148 

045 

475 

00:00 

2 

N120  W72 

135 

350 

00: 3C 

3 

N84  El 2 

070 

260 

01:00 

4 

848  El 2 

255 

400 

01:30 

5 

N120  W96 

090 

560 

02:00 

6 

N72  E72 

270 

250 

03:00 

7 

8136  W12 

315 

200 

06:00 

8 

836  W48 

060 

325 

07:00 

9 

836  E84 

230 

150 

09:00 

10 

N108  E46 

270 

500 

12:00 

11 

N84  W96 

170 

410 

14:00 

12 

N36  E144 

260 

350 

15:00 

13 

3136  El 20 

320 

370 

17:00 

14 

N180  E24 

210 

230 

16:00 

15 

N12  W168 

100 

420 

20:00 

The  problem  was  prepared  in  four  equivalent  forms  by 
rotating  the  original  problem,  described  above,  90°  clockwise  in 
three  successive  steps.  This  procedure  kept  the  time  and  spatial 
relations  within  the  problem  constant,  but  eaoh  presentation  was 
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S  NOTE 

Itolics  designate  Target  Number. 
Parenthesis  designate  Time  of 
Appearance. 

Schematic  Representation  of  Problem  I  -  Full 

Target  Tracks 


reoriented  so  that  It  was  perceptually  different  each  time. 

This  perceptual  difference  was  later  confirmed;  none  of  the  subjects 
guessed  that  they  had  been  working  with  essentially  the  same  problem. 
The  four  replications  of  the  experiment  have  been  designated  as 
Experiments  11.11,  11.12,  11.13  and  11.14,  respectively.  Two 
subjects  were  tested  at  a  time  on  four  successive  days. 

5.  Subjgsta 

The  four  experimental  subjects  studied  here  were  naval 
officers,  ranking  from  Ltjg.  to  Lcdr.,with  shipboard  or  airborne 
CIC  experience.  Before  the  experiment,  the  subjects  became 
thoroughly  fa  . -liar  with  the  APA/?6  gear,  with  which  they  had  had 
no  previous  experience.  The  subjects  were  given  one  preliminary 
practice  problem  for  familiarization  with  the  problem,  instructions, 
and  procedures  involved. 

It  was  necessary  to  have  a  record  of  both  the  subjects' 
responses  and  the  problem  as  it  appeared  to  them,  in  order  to 
effect  the  desired  measurements.  These  data  were  obtained  by 
photographing  the  scope  picture  at  every  sweep,  by  complete  record¬ 
ings  of  subjects'  verbal  reports,  and  by  direct  observation  of  the 
subjects'  plotting  behavior. 

The  visual  presentation  on  the  scope  face  was  recorded 
on  a  Fairchild  camera  which  was  set  to  take  a  picture  of  every 
sweep  that  was  presented.  This  meant  that  six  frames  were  exposed 
every  minute  giving  a  complete  record  of  how  the  problem  appeared 


at  any  particular  sweep  during  the  problem.  From  such  s  record 
the  actual  bearing  and  course  to  the  nearest  degree,  and  range 
and  speed  to  the  nearest  mile  could  be  measured  by  projecting  the 
image.  This  was  necessary  because  there  is  always  some  error  in 
both  setting  the  problem  into  the  target  generator  as  well  as  in 
the  generator  itself.  However,  the  camera  records  the  problem 
as  it  actually  appeared  on  the  scope  face  and  so  becomes  the  most 
accurate  record  of  the  problem. 

The  second  method  of  recording  data  utilized  voice 
recordings.  The  verbal  responses  of  each  subject  were  recorded 
on  a  wire  or  tape  recorder.  These  verbal  reports  were  concerned 
with  the  detection  of  new  targets,  their  range,  bearing,  speed  and 
course.  Also  on  those  recordings,  a  time  signal  (the  work  "Mark") 
served  to  indicate  the  start  of  the  problem  and  th8  insertion  of 
each  new  target.  This  provided  a  time  base  on  the  voice  recording 
from  which  the  various  measures  were  made,  when  the  recordings  were 
played  back. 

Finally,  the  plotting  behavior  was  observed  by  an  ex¬ 
perimenter  inside  the  hull  who  recorded  each  subject’s  plotting 
responses  every  ether  minute.  In  this  way  a  50  per  cent  time 
sample  was  obtained  on  each  subject. 

Since  the  primary  data  were  collected  first  on  camera 
film  and  voice  recorders,  it  was  necessary  to  transcribe  the 
data  to  obtain  a  record  in  written  form.  This  was  accomplished 
by  viewing  the  film  record  of  the  scope  face  on  a  Recordak 
micro  film  reader.  By  appropriate  measurements,  the  range, 
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bearing,  speed  and  course  of  the  targets,  as  they  actually  appeared 
on  the  scope,  could  be  determined.  The  voice  recordings  were 
transcribed  by  playing  back  the  records  and  writing  down  the 
verbal  reports  of  the  subjects.  These  reports  were  timed  to  a 
tenth  of  a  minute.  Since  one  tenth  of  a  minute  corresponds  to 
less  than  one  sweep  revolution  on  the  scope  face,  the  accuracy  of 
the  data  is  within  one  sweep  revolution. 
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IV.  RESULTS 


The  following  sections  present  the  basic  data  of  this  study. 
The  results  are  presented  in  four  sections  corresponding  to  the 
functions  studied;  detection  time,  accuracy,  reporting  and  plot¬ 
ting. 

1.  Detection  Time  of  New  Targets 

The  data  presented  In  Table  2  and  Fig.  2  represent  the 
mean  detection  time  for  any  given  target  as  influenced  by  the  num¬ 
ber  of  targets  previously  Introduced  on  the  scope  face. 


Table  2 

Detection  Time  of  a  New  Target* 


Number  of  Tar¬ 
gets  on  Scope 

Mean  Detection  Time 
of  the  New  Target 

Standard  Error 
of  the  Mean 

Standard 

Deviation 

0 

.1 

.01 

.1 

1 

.1 

.01 

.1 

2 

.1 

.01 

3 

.2 

.01 

.2 

4 

.2 

.01 

.2 

5 

.1 

.01 

.2 

6 

.3 

.02 

.3 

7 

.3 

.03 

.4 

8 

.4 

.02 

9 

< 

•  y 

•°3 

10 

1.0 

.06 

.7 

11 

.9 

.05 

.7 

12 

1.^ 

.08 

.9 

13 

1.3 

.08 

.9 

14 

1.4 

.43 

1.4 

♦All  data  are  in  minutes. 


The  data  presented  represent  the  mean  detection  time  for  different 
numbers  of  targets  already  on  the  scope  for  all  four  subjects, 
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FIG.  2  -  Detection  Time  as  a  Function  of  the  Number  of 

Targets  on  the  Scope  Face 
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NUMBER  OF  TARGETS  ON  SCOPE  FACE 


with  each  subject  having  been  tested  four  times.  The  standard 
deviations  and  standard  errors  are  also  presented. 


2.  Accuracy  of  Estimations  of  Target  Characteristics 

The  accuracy  of  the  estimation  of  target  characteristics 
is  revealed  by  the  errors  made  in  judging  target  range,  bearing, 
course  and  speed.  These  errors  were  found  by  taking  the  difference 
between  the  reported  positions  and  the  actual  or  true  positions  of 
the  targets.  Errors  may  be  considered  from  two  points  of  view: 
relative  error  and  absolute  error.  Relative  error  considers  the 
magnitude  and  sign  of  the  deviations;  absolute  error  considers 
the  magnitude  only.  The  mean  relative  error  gives  a  measure  of 
the  constant  error  of  over  or  under  estimation.  The  standard 
deviation  of  the  relative  errors  is  an  inverse  measure  of  the 
precision  of  estimation.  The  median  absolute  error  is  a  measure 
of  the  accuracy  of  the  estimates  without  respect  to  the  direction 
of  the  error. 

The  errors  which  resulted  under  the  conditions  of  this 
experiment  are  reported  in  Tables  3,  4  and  5.  The  data  represent 
average  values  for  sll  subjects  for  all  runs, 
a.  Bearing  Errors. 

The  frequency  of  the  degrees  of  error  for  relative 
and  absolute  errors  is  given  in  Table  3  and  Fig.  3. 

It  was  found  that  the  mean  relative  bearing  error  was 
0.53°  with  a  standard  deviation  of  2.58°;  the  median 
absolute  bearing  error  was  1*76°.  Ninety-five  per  cent 
of  the  errors  did  not  exceed  6 ,25°»  Absolute  bearing 
error  as  a  function  of  number  of  targets  on  the  scope 


is  given  in  Fig.  6 
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Table  3 

Errors  of  Bearing  Estimation 


Error 

in 

Degrees 

Frequency  of  Errors 

Relative  Errors 

Absolute  Errors 

10 

10 

16 

9 

8 

11 

8 

11 

16 

7 

18 

27 

6 

20 

27 

5 

58 

75 

4 

62 

85 

3 

117 

192 

2 

201 

339 

1 

275 

478 

0 

538 

538 

-1 

203 

-2 

138 

-  *5 

75 

-4 

23 

-5 

17 

-6 

7 

Median  1.8 

-7 

9 

-8 

5 

«3  3.0 

-9 

3 

-10 

6 

Qj  .8 

Mean  0.5 

Interquartile 

Range  2.2 

Standard 

Deviation  2.6 

Total  Errors  1804 

b.  Range  Errors. 

Range  errors  were  treated  similarly.  The  frequency 
distribution  of  range  errors  with  respect  to  sign  is 
given  in  Table  4.  The  relative  mean  range  error  is  1.85 
miles,  and  the  standard  deviation  is  2.17.  The  absolute 
range  error  frequency  distribution  is  given  in  Table  4 
and  Fig.  4.  The  median  absolute  range  error  is  2.49  mile 
Ninety-five  per  cent  of  the  range  estimates  showed  an 
error  of  less  than  6  miles.  The  relationship  between 
absolute  range  errors  and  the  number  of  targets  on  the 

scope  is  presented  in  Fig.  6* 
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ERROR  IN  DEGREES 

FIG  3-  Absolute  Bearing  Errors 

c.  Course  Errors. 

The  distribution  of  relative  course  errors  is  given 
in  Table  5*  The  mean  relative  course  error  is  1.1 3° * 
and  the  standard  deviation  is  11.95°»  The  absolute 
course  error  frequencies  are  given  in  Table  5  and  Fig.  5* 
The  median  absolute  course  error  is  5.83°,  and  95  per 
cent  of  the  errors  are  less  than  28. 5° • 

d.  Speed  Errors. 

The  estimations  of  the  speed  of  the  targets  were  in 
such  gross  error  that  the  data  were  not  given  extensive 
statistical  treatment.  The  average  error  in  speed  esti¬ 
mation  showed  an  under  estimation  of  130  knots  per  hour. 
The  range  of  absolute  errors  in  speed  estimation  extended 
from  zero  to  350  knots  per  hour. 
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Table  4 

Errors  of  Range  Estimation 


Error 

In 

Miles 

Frequency  of  Errors 

Relative  Errors 

Absolute  Errors 

10 

7 

9 

9 

4 

6 

8 

16 

19 

7 

12 

17 

6 

28 

32 

5 

70 

76 

4 

143 

150 

3 

282 

298 

2 

473 

499 

1 

291 

3*7 

0 

253 

253 

-1 

56 

-2 

76 

.-3 

16 

-4 

7 

-5 

6 

-6 

4 

Median  2.5 

-7 

5 

-8 

3 

Q,  3.6 

-O 

2 

-10 

2 

Ql  1*5 

Mean  1*9 

Interquartile 

Range  2.1 

Standard 

Deviation  2.2 

Total  Errors  1706 

3*  Number  of  Reports  Made 

A  report,  as  defined  here,  is  any  information  unit  on 
a  target,  such  as  bearing,  range,  course,  or  speed.  The  data  were 
analyzed  with  respect  to  the  number  of  reports  made  within  a 
minute  interval  as  the  experiment  progressed.  The  findings  are 
presented  in  Fig.  ?.  Since  the  number  of  targets  on  the  scope  is 
also  a  function  of  time,  this  variable  has  also  been  plotted  on 
the  aosclssa. 
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PERCENT  OF  ERROR  AS  GREAT  AS  PERCENT  OF  ERROR  AS  GREAT  AS 

AMOUNT  SHOWN  ON  ABSCISSA  AMOUNT  SHOWN  ON  ABSCISSA 
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FIG.  4-  Absolute  Range  Errors 


FIG  5  -  Absolute  Course  Errors 
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Table  5 

Errors  of  Course  Estimation 


Another  way  of  inspecting  the  number  of  reports  made 
is  by  considering  the  time  between  successive  reports  for  a  given 
target.  Targets  1,  5,  and  10  were  considered  as  a  sample.  The 
time  between  successive  reports  on  these  targets,  considered  with 
respect  to  the  number  of  targets  on  the  scope  face,  is  presented 
in  Fig.  8. 

4.  Plotting  Behavior 

From  the  time  samples  taken  cf  the  olotting  behavior  of 
all  subjects  on  all  experiments,  the  mean  number  of  plots  per 
minute  was  calculated.  These  data,  plotted  as  a  function  of  run¬ 
ning  time,  are  presented  in  Fig.  7.  The  grand  mean  number  of  plots 
per  minute  is  5.26. 
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DEGREES 


FI6. 6-  Bearing  and  Range  Errors  as  a  Function  of  the 
Number  of  Targets  on  the  Scope  Face 


SldOd3d  30  d38WnN  NV3W 

FIG.  7  —  Number  of  Reports  per  Minute  and  Number  of  Plots 
per  Minute  as  a  Function  of  Operating  Time 
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FIG.  8  -  Time  Between  Successive  Reports  on  a  Given 
Target  as  a  Function  of  the  Number  of  Targets  on 

the  Scope  Face 
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V.  DISCUSSION 


1.  Detection  Time  of  New  Targets 

In  an  AEW  operation,  the  primary  function  of  CIC  is  the 
detection  of  new  targets.  The  problem  which  was  raised  earlier 
was:  How  is  this  detection  time  affected  by  the  presence  of  a 
number  of  targets  and  their  plotted  tracks  already  on  the  scope? 

The  data  show  very  clearly  that  as  the  number  of  targets  already 
on  the  scope  increases,  the  detection  time  for  a  new  target  also 
increases  as  a  positively  accelerating  function.  The  more  targets 
on  the  scope,  the  lonper  it  takes  to  detect  a  new  target. 

Stated  in  terms  of  the  difficulty  of  the  subjects'  task, 
these  data  signify  that  as  the  number  of  targets  present  on  the 
scope  face  increases,  it  becomes  progressively  more  difficult  to 
perceive  and  report  the  appearance  of  a  new  target.  On  the  aver¬ 
age,  it  takes  0.1  min.  to  detec  and  report  a  new  target  appear¬ 
ing  on  a  previously  blank  scope;  0.14  min.  to  pick  up  and  report 
a  new  target  in  the  presence  of  four  ''old"  ones;  and  1.35  min.  to 
observe  and  report  an  additional  target  when  14  targets  are  already 
on  the  face  of  the  CR  tube.  The  saturation  level  of  an  operator 
with  respect  to  detection  time  can  only  be  considered  in  relation 
to  certain  criteria  of  operational  performance.  If  the  upper 
time  limit  required  for  target  detection  and  reporting  is  speci¬ 
fied,  the  conditions,  in  terms  of  number  of  targets  on  the  scope, 
under  which  the  limit  may  be  met,  can  be  found  in  Fig.  2.  If  a 
detection  time  of  one  minute  is  required,  no  more  than  11  targets 
can  appear  on  the  scope  face  when  a  single  operator  searches  with  a 
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centered  presentation.  If  there  are  14  targets  on  the  scope 
face,  it  will  take  1.35  win.  to  detect  a  new  target. 

While  the  subjects  reported  that  they  might  be  able  to 
handle  more  than  15  targets,  the  work  load  was  high,  putting  the 
operators  under  a  great  deal  of  stress.  Qualitative  analysis  of 
the  data  indicates  that  with  more  than  ten  targets  on  the  scope 
a  certain  amount  of  confusion  arises  with  respect  to  detecting 
and  numbering  the  targets.  Target  13  may  be  detected  and  desig¬ 
nated  prior  to  target  12:  and  in  addition,  target  14  may  be  con¬ 
fused  with  target  11,  and  so  on. 

2 .  Errors  in  Estimation  of  Target  Characteristics 

Information  concerning  errors  in  the  estimation  of  range, 
bearing,  course  and  speed  of  targets  is  another  area  of  investiga¬ 
tion  which  might  give  some  insight  into  the  capabilities  and  po¬ 
tentialities  of  the  men  and  machines  involved  in  the  system  being 
studied.  Two  aspects  of  this  problem  are  of  particular  interest. 
First,  it  is  important  to  learn  what  the  errors  are  in  the  direct 
visual  estimation  of  target  characteristics  when  the  judgment? 
are  essentially  unaided.  The  data  obtained  under  tlv's  condition 
may  then  serve  as  a  standard  against  which  to  evaluate  experimental 
conditions  which  include  the  use  of  newly  developed  operational 
aids,  such  as  the  range  and  bearing  strobes  incorporated  into  the 
APA/56. 

Second,  the  question  arises  as  to  whether  a  point  of  satu¬ 
ration  may  also  be  indicated  by  a  decrease  in  accuracy  in  report¬ 
ing  the  various  target  attributes  under  study?  Such  saturation 


would  be  another  Indication  of  a  limitation  in  the  capacities  of 
a  single  operator  and  his  associated  equipment, 
a.  Range  and  Bearing  Accuracy. 

Consideration  of  the  data  (Tables  3  and  4,  and  Figs. 

3  and  4)  on  the  accuracy  of  range  and  bearing  shows  that 
the  errors  of  estimation  are  small  and  stable.  From  the 
results  it  may  be  expected  that  95  times  out  of  100  the 
errors  in  range  will  not  exceed  6  miles,  and  6  degrees  in 
bearing  in  either  direction.  The'  negligible  si7e  of  the 
errors  of  the  estimations  and  the  stability  of  the 
Judgments  would  seem  to  indicate  that  under  certain 
conditions  it  might  be  entirely  feasible  to  give  such 
estimations  within  certain  limits  of  accuracy  without  the 
use  of  mechanical  or  electronic  devices.  This  would  make 
for  both  a  saving  of  time  in  obtaining  target  information, 
and  a  simplification  in  the  operation  and  design  of  the 
gear. 

The  effect  of  increasing  the  number  of  targets  on  the 
scope  on  the  accuracy  of  estimations  of  range  and  bearing 
is  indicated  in  Fig.  6.  These  data  indicate  that  increasing 
the  number  of  targets  has  no  deleterious  effect  on  these 
activities.  In  fact,  there  appears  to  be  some  improvement. 

It  would  appear  that  these  Judgments  do  not  suffer  when 
the  work  load  is  increased.  No  saturation  limit  with  respect 
to  these  functions  can  be  said  to  have  been  reached  under 
the  conditions  of  this  experiment. 


b.  Course  Accuracy. 

Without  respect  to  the  direction  of  the  error,  the 
median  course  error  was  about  7.5°.  This  is  considerably 
greater  than  estimation  of  bearing  given  above.  The  vari¬ 
ability  of  these  estimations  was  also  much  greater:  the 
sigma  of  the  relative  course  errors  being  11.95°.  It 
would  seem,  then,  that  the  accuracy  of  unaided  estimation 
of  course  may  not  give  the  precision  necessary  for  certain 
types  of  operations,  e.g.,  vectoring  CAPs,  but  may  be 
adequate  for  work  in  which  rougher  and  less  reliable 
estimates  are  satisfactory. 

c.  Speed  Accuracy. 

Inspection  of  the  raw  data  reveals  gross  errors  in 
speed  estimation.  For  the  most  part,  the  speed  of  the 
targets  was  underestimated.  With  respect  to  this  function, 
direct  estimation  is  too  unreliable  and  inaccurate  without 
some  aid  such  as  the  three  minute  rule.  Another  study,  to 
be  reported  elsewhere,  indicates,  however,  that  accurate 
direct  estimates  of  speed  may  be  possible  with  some  training 
of  the  observers. 

3.  Number  of  Reports  Made 

Since  the  task  of  an  AEW  combat  information  center  is  not 
complete  until  the  information  which  has  been  collected  is  dis¬ 
seminated,  it  is  of  great  interest  to  learn  something  about  the 
reporting  behavior  of  the  subjects. 


JHHKSBSSi. 

Graphic  demonstration  of  the  data  concerning  this  be¬ 
havior  is  presented  in  Fig.  7»  The  shape  of  the  curve  resembles 
closely  the  general  shape  of  a  work  or  fatigue  curve.  The  os¬ 
cillating  nature  of  the  curve  seems  to  indicate  periods  of  high 
work  output  followed  by  periods  of  low  output  or  relative  rest 
periods.  The  curve  oscillates  around  eleven  reports  per  minute. 

The  subjective  reactions  of  the  operators  indicate  that 
they  were  under  considerable  stress  and  tension,  and  quite  fatigued. 
Qualitatively,  it  was  of  interest  to  note  that  the  initial  dec¬ 
rement,  occurring  between  the  10th  and  16th  minute  interval,  was 
accompanied  by  confusion  in  detecting  the  targets  in  the  correct 
successive  order.  In  the  opinion  of  the  experimenter,  additional 
targets  and  lengthening  operation  time  would  have  seriously  dis¬ 
rupted  the  performance  of  the  operators. 

Since  the  sweep  makes  six  revolutions  per  minute,  and, 
on  the  average,  eleven  reports  were  made  per  minute,  it  appears 
that  slightly  less  than  two  reports  were  made  per  sweep.  This 
usually  manifested  itself  as  a  Joint  report  on  range  and  bearing 
or  course  and  speed  of  a  given  target.  From  this  information  it 
may  be  deduced  that  when  there  are  considerably  more  than  6  tar¬ 
gets  on  the  scope,  each  target  cannot  be  reported  as  often  as 
once  per  minute. 

In  order  to  check  this  deduction,  the  data  were  considered 
from  another  point  of  view.  The  time  between  successive  reports 
for  a  given  target  was  tabulated  and  considered  with  respect  to 
the  number  of  targets  on  the  scope  face.  These  dsta  are  presented 
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in  Fig.  8.  It  can  readily  be  seen,  as  was  to  be  expected,  that 
as  the  number  of  targets  to  be  reported  increase,  the  time  between 
successive  reports  on  a  given  target  increased.  When  there  are 
8  targets  to  be  reported,  a  criterion  of  one  report  per  minute 
of  each  target  cannot  be  met. 

4.  Plotting  Behavior 

Since  it  is  necessary  to  have  not  only  current  information 
on  the  target,  but  also  to  have  some  past  history  in  order  to 
estimate  the  course  and  speed  of  the  target,  plotting  of  the  target 
is  necessary  to  keep  a  record  of  that  target.  Plotting  behavior  was 
consequently  observed  to  get  some  information  on  this  function.  The 
results  showed  very  clearly  that  the  differences  among  operators 
was  significant;  some  individuals  plotted  consistently  with  a 
frequency  as  low  as  3  plots  per  minute  and  others  as  high  as  7 
plots  per  minute.  On  the  average,  5»26  plots  were  made  per  minute 
for  all  subjects.  This  means  that  about  one  plot  was  made  for  every 
scan  of  the  radar  sweep.  This  behavior  was  very  consistent  and 
showed  no  decrement  with  time  as  is  indicated  by  Fig.  7. 


